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CONFERENCE PUBLICATION 


NASA TECHNICAL INTERCHANGE MEETING (TIM): ADVANCED TECHNOLOGY 
LIFECYCLE ANALYSIS SYSTEM (ATLAS) TECHNOLOGY TOOL BOX 


On November 8 th through 10 th , 2004, Marshall Space Flight Center (MSFC) hosted Technical 
Interchange Meeting (TIM) in Huntsville, Alabama. The TIM focused on the Technology Tool 
Box (TTB), a technology database developed for use by Advanced Technology Lifecycle 
Analysis System (ATLAS). ATLAS models use TTB data to model the impact of investing in 
different technologies. 

The overarching goal of the three-day meeting was to improve the TTB by increasing the 
quantity and quality of technical, cost, and programmatic data on different technologies. This 
report describes the results of the November meeting, and also provides background information 
on ATLAS and the TTB. 

1. INTRODUCTION TO ATLAS AND THE TECHNOLOGY TOOL BOX 


Currently under development, ATLAS provides a capability to evaluate technology portfolios 
that support future space exploration. Control software within ATLAS manages a collection of 
Excel workbooks, each describing an element within an overall space exploration architecture. 
Within the context of ATLAS, a collection of space transportation systems, supporting 
infrastructure, and scientific payloads constitutes a space architecture. An analyst can assemble 
a complete space architecture by combining modeled elements and specifying parameters and 
technologies to be employed in those elements. For the given set of parameters and 
technologies, the analyst can display system performance parameters such as mass and cost. 
Thus, ATLAS provides strategic planners with a decision support tool to evaluate potential 
technology portfolios. 

ATLAS models rely on data about the different technologies drawn from the TTB. The TTB 
contains technology data from within NASA as well as data gathered from outside sources. 

Presently, the TTB exists as an Excel workbook but work has begun on a web-accessible 
version. On the first day of the TIM, Daniel O’Neil and Lamonte Dent demonstrated the 
prototype web version of the TTB depicted in Figure 1 . 

The record for each technology includes approximately two dozen performance metrics (such as 
mass and power required), operational metrics (such as operational lifetime and mean time 
between failures) and programmatic metrics (such as technology readiness level, index value 
reflecting development risk, and development timelines). Furthermore, the TTB also contains 
projected future values of these metrics. These projected technology metrics allow ATLAS to be 
used as a tool to rapidly evaluate and compare the potential cost and performance benefits over 
time of various technology investment scenarios. 




Figure 1 A web-accessible Technology Toolbox (TTB), which will allow for easy updating 

and additions, is under development. This figure illustrates an example of the user 
interface for the TTB. 
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2. MEETING OVERVIEW AND OBJECTIVES 


The overarching goal of the three-day meeting was to improve the TTB by increasing the 
quantity and quality of technical, cost, and programmatic data on different technologies. To this 
end, TIM participants were asked to provide specific data and comment on data provided by 
others; to discuss underlying definitional and interface issues with regard to the technology tool 
box and related tools, and to address architecture and system issues for the TTB, ATLAS, and 
the related Simulation Based Acquisition (SBA) system currently under development by NASA 
Exploration Systems Mission Directorate (ESMD). 

The objectives of the TIM were outlined in a plenary session on the first day and addressed in 
detail in breakout working groups on the second day. The results, accomplishments, and 
achievements of the working groups were briefed and discussed at a plenary session on the third, 
and final, day of the meeting. 


2.1 Opening Plenary Session 

ATLAS and its components (such as the TTB) are part of the Exploration Systems Research and 
Technology (ESR&T) Program of the NASA Headquarters Exploration Systems Mission 
Directorate (ESMD) at NASA Headquarters. 


ESR&T/ESMD has a dedicated element focusing on analytic studies and tools for advanced 
exploration concepts. Doug Craig, the Program Element Manager for this activity (Advanced 

Studies Concepts and Tools, 
or ASCT), kicked off the 
TIM during Monday’s 
plenary with a perspective 
from NASA Headquarters. 
He presented an overview of 
the ESR&T Program and 
ASCT within it, providing 
guidance and background for 
the discussions that followed 
over the next two days in 
each of the working groups. 



Mr. Craig described the 
ASCT investment portfolio 
assessment process and the 
context for ATLAS and the 
TTB (depicted in Figure 2). 


Figure 2 Doug Craig introduced the participants to ASCT and defined 
ATLAS and the TTB’s role within ASCT and ESR&T 
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In this process, Advanced Concept Design activities generate requirements for ATLAS 
workbook system models. Analysts use ATLAS to configure systems and select technologies to 
meet mission campaign objectives. Other tools for detailed risk assessment will use the system 
configurations, technology portfolios, and technology data provided by ATLAS and the TTB. 
Other technology databases will provide narrative descriptions, pictures, and technology 
development history. Links to a common Work Breakdown Structure (WBS) and other key data 
fields will provide a capability to correlate records among the TTB and other technology 
databases. 


2.2 Working Groups 

The hands-on work of the TIM was conducted through breakout working groups on day two. 

The diverse participant list of the TIM allowed for an open dialogue and a sharing of ideas for 
the database between industry and NASA personnel. Participants represented the following 
teams and organizations: 

■ Winning Exploration Systems Research and Technology (ESR&T) Intramural Call for 
Proposals (ICP) teams, 

■ Contractors from Exploration Systems Mission Directorate (ESMD) Capabilities Office 
Concept Exploration and Refinement (CE&R) teams, 

■ The Advanced Studies Concepts and Tools (ASCT) Leadership team, 

■ The SBA development team, 

■ Local contractors researching historical data for the TTB, 

■ The Future Concepts Office within the MSFC Space Systems Programs & Projects 
Office, and 

■ The ATLAS project team, including The Tauri Group, which facilitated data collection 
and working group reports in addition to participating in technical discussions. 

Each of the forty-two technology experts participating in the TIM was assigned to one of three 
groups: Technology Data Development and Collection; Architectures and Systems; and, 
Database Definitions and Interfaces. The groups conducted detailed discussions and addressed 
specific technical goals. 

The Technology Data Development and Collection (TDDC) working group (chaired by Monica 
Doyle and Jamie Esper) The group was charged to review and come to consensus on technology 
data provided by participants. The group was also asked to expand on and document the data, by 
defining growth scenarios for selected data sets and developing a completion plan for marginal 
data sets, with likely assignments of individuals to provide certain data. 

The Architectures and Systems (A&S) working group (chaired by Harvey Feingold and Carissa 
Christensen) was charged to define what is needed for successful ATLAS modeling and analysis. 
The group was directed to discuss ATLAS existing and planned models. The discussion 
encompassed the technology data the models currently use, additional types of data they should 
use in future versions, and the interplay and relationship of ATLAS models to the TTB and the 
SBA. Based on these discussions, the group was asked to identify strategies for integrated use of 
TTB. 
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The Database Definitions and Interfaces (DD&I) working group (chaired by Doug Craig) was 
charged to develop an analysis plan for ESR&T, assessing the possibility for other tools used by 
ESR&T to employ the TTB and/or ATLAS. The group’s activities constituted important pre- 
planning for a future tools workshop to be hosted by Ames Research Center (ARC). 
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3. MEETING FINDINGS AND ACCOMPLISHMENTS 


The findings and accomplishments of each of the three working groups are summarized below. 
Each of the groups developed a briefing package that reported out its discussion and 
recommendations. Appendix B contains the complete out-briefs from the working groups. 

3.1 Technology Data Development and Collection (TDDC) Working Group 

The TDDC working group addressed key technology data issues in its breakout session on day 
two. Jaime Esper and Monica Doyle led the discussion among 20 participants in the TDDC 
working group. This working group successfully 
collected 27 data sets in 16 areas of the Advanced 
Systems Technology Research and Analysis (ASTRA) 
work breakdown structure (WBS). The group 
specifically discussed in detail WBS elements 2.2.1 
Space Solar Power and 2.6.1 Earth-to-orbit 
transportation during the breakout session. Figure 3 
identifies the number of data sets collected in each of 
these areas at the three-digit WBS level. 

The TDDC working group also fonnulated 
recommendations for improving the data collection 
process, refining definitions of data fields within the 
TTB, and defining operating assumptions to provide a 
context for technology data. 


TDDC Findings and Accomplishments 

v' 

Collected 27 sets of data in 16 
technology areas 

✓ 

Developed strategy for handling 
contradictory data submissions 

s 

Outlined processes to improve 
data collection efforts 

s 

Discussed export control and 
proprietary data issues 

s 

Improved definition and 
understanding of goal and 
threshold values 

s 

Identified potential for 
technology maturation paths 


The working group addressed the challenging 
question of how the TTB should handle different 
types of sensitive data, specifically company- 
proprietary data and data that might be subject to 
export controls. 

In considering whether the TTB could omit 
proprietary data to avoid data protection problems, 
participants concluded that the utility of the TTB 
requires competing companies to share data. 
Participants reported that their companies were 

most concerned about sharing budget forecasts 
required to achieve certain performance levels in a 
specific timeframe; they said that budget projections 
were more sensitive than forecasts of performance and operations metrics. Participants also 
generally agreed that it would be helpful to avoid specifically identifying a company as a source 
of data, to ensure data integrity. 



Participants discussing specific technologies in 
the TDDC Working Group 
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Industry participants also raised concern about export control restrictions, because the 
aggregation of data from many sources in the database might result in its being subject to export 
controls, while specific data provided by an individual contributor was not. From a security 
perspective, the whole of the database could be greater than the sum of its parts. An 
unauthorized release of a single record may not be a problem; however, the release of the entire 
database would a problem. Individual data contributors were concerned about their ability to 
follow export control procedures if they contributed data when they did not have control over the 
whole database. The group formulated recommendations for handling these issues by 
controlling access at the database, record, and field levels. It was also generally agreed if this 
process were followed it would be easier to collect data. 

Other discussions involved the definitions of several important and complex metrics: technology 
performance goals, threshold values, and technology readiness level (TRL) forecasts. The terms 
“goal” and “threshold”, to describe Technology 
Development progress, were considered 
difficult to define precisely by many of the 
participants. One suggestion for defining these 
terms was to replace these parameters or 
categories with mean and deviation. (In the near 
tenn, technologist could assume a normal 
probability distribution. Later versions of the 
TTB could offer a selection of pro-bability 
distributions as a drop-down menu field.) The 
group also sought additional precision in 
estimates of technology readiness level. 

Several technologists stated that a TRL path 
indicating the relative focus on tech-nology 
maturation versus pure research should be 
specified before fore-casting performance and 
operations metrics. (In a technology maturation 
path, the TRL increases while the performance 

values remain constant. In a pure research path, 
the perfonnance increases while the TRL 
remains constant.) It was noted that this could 
add structure to the data collection efforts. 

Finally, the group raised the question of whether ESMD could characterize its future 
development spirals in a manner that indicated where a certain level of technology maturity 
(typically TRL 6, at which the technology has been demonstrated) would be needed. 



Received Data Sets 


Figure 3 Number of data sets received 
in technology areas 
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3.2 Architectures and Systems (A&S) Working Group 


The A&S working group addressed a range of issues including: modeling and analysis needs in 
ATLAS and the TTB; planned models and architectures to be used with the TTB, ATLAS, and 
SB A; and technical data requirements driven from the architectures and system needs. Harvey 
Feingold and Carissa Christensen led discussions among the fifteen participants in the working 
group, which came to findings related to context, the user community, system functions, and 
optimization. 

The context, primary functions, and uses/users for 
ATLAS were discussed to provide a shared 
understanding of the current system. ATLAS, it was 
agreed, provides a quick-look capability to figure out 
what makes sense. With ATLAS, a strategic planner 
can compare technologies, conduct top-level 
architectural trade studies, and plot “living forecasts” 
from the TTB. ATLAS is not to be considered as or 
used as an engineering design tool. The functions of ATLAS can be grouped into: technology 
payoff calculations and a tool to compare technology, mission, and campaign strategies. 
Presently, the ATLAS user community consists management and designated representatives of 
the ESR&T program office. 

The group considered what user groups should be targeted as ATLAS moves forward. Potential 
users identified included: NASA Headquarters program managers, strategic planners, program 
planners, engineers, technologists, and perhaps students as a teaching tool. The group noted that 
ATLAS models require a full understanding of the capabilities and limitations of the models; 
both technologists and system model developers were concerned about the possibility of garbage 
in/garbage out. In fact, the consensus of the A&S working group was that the TTB (rather than 
specific models) might be the most useful part of ATLAS for a broad base of users. 


The working group reviewed plans for optimization 
functions in ATLAS; the team noted that the ATLAS 
proposal for future activities includes this function as 
part of the later development. The team recommended 
that figures of merit to use as goals for optimization 
include cost, mass, and safety or risk and that items to 
vary in the optimization function include technology, 
systems, and architectures. 


Several participants in the A&S working group asked, 
“What is the relationship between ATLAS and the Simulation Based Acquisition (SBA) 
project?” which led to further discussion. Doug Craig addressed this question by explaining that 
both systems will work towards a common data dictionary or ontology, the SBA supports the 
strategy-to-task activity, and SBA will support a more detailed level of analysis than ATLAS. 
The two projects will share tools and develop compatible data file formats. For example, the 
SBA will develop risk analysis tools at the technology and system level. The ATLAS could 
provide data to or receive data from these risk analysis tools. 



The A&S working group addressing big- 
picture ATLAS and TTB issues 


A&S Findings and Accomplishments 

✓ 

Clarified the context for ATLAS 

S 

Identified the user community for 


ATLAS and the TTB 

S 

Outlined system functions 

Y 

Discussed optimization functions 
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3.3 Database Definitions and Interfaces (DD&I) Working Group 

Doug Craig and Daniel O’Neil led a discussion among nine participants in the DD&I working 
group. Accomplishments included: planning for convergence of the ATLAS and SBA 
technology databases, pre-planning for the future tools workshop, planning architecture 
development and campaign analysis activities, and defining terms used across ASCT. 

During pre-planning for the tools workshops working 
group members were assigned actions to document the 
process and products associated with ASCT activities 
Advanced Concepts, Technology Databases, Campaign 
Analysis, Risk Analysis, and Data Visualization. These 
plans will identify potential opportunities for integration 
of SBA and ASCT tools. Most of the group’s participants 
were tasked with documenting process and products in 
preparation for the tools workshop. 

Discussions on architecture development and campaign 
analysis activities addressed the diagram in Figure 2. In 
Figure 2, the box in the upper left comer identifies incoming requirements from the ESMD 
Requirements Directorate and Project Constellation. These requirements will include a Point-of- 
Departure architecture. The Advanced Concept Design team will analyze the requirements and 
develop alternative architectures with system concepts that emphasize technologies under 
development by ESR&T. To assemble these architectures within ATLAS, the Advanced Concept 
Design and ATLAS teams will work together to create Excel workbook models. A campaign 
assessment team will use ATLAS to evaluate the impact of technologies to the lifecycle costs of 
the architectures. 

The working group specifically discussed several terms that are used throughout ASCT, and 
came to agreement on clear definitions. The terms defined by the DD&I working group were: 

• Architecture , a collection of systems that provide the capability to perfonn one or more 
types of missions. 

• Campaign analysis , evaluating benefits across a sequence of missions and systems. This 
analysis involves determination of architecture sensitivities to technology portfolios. 
Campaign analysis will support the ASCT Advanced Study teams, answer action items 
received by the ESR&T program office, and serve as an integral part of the annual 
technology assessment and investment process. Through campaign analysis, the ASCT 
will verify and validate technology data. 

• Point-of-departure architecture , a reference for benefit assessments produced by NASA 
and the Concept Exploration and Refinements Teams. 

• Ontology , the organizational structure of the campaign, architecture, systems, and 
technology data. 

• Threshold value , the minimum achievable value within a range of technology 
parameters. 


DD&I Findings and Accomplishments 


/ Planned for convergence of 
ATLAS and SBA databases 
•/ Pre-planning for future tools 
workshop 

s Outlined architecture 

development and campaign 
analysis activities 
✓ Defined several cross-cutting 
ASCT terms 
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The DD&I working group also discussed and 
came to agreement on an analysis plan for 
ATLAS and the TTB. Steps in the plan 
include: defining data requirements, 
developing a centralized repository of 
technology data, determining a data migration 
path, and decide on a risk analysis tool. 

Specifically the team decided that a data 
verification and validation team within ASCT 
will define the requirements for the TTB data 
collection. These requirements will define 
the data pedigree within the database. The 
centralized repository of technology data 
would provide controlled access to the 
technology records, while allowing record 
owners to provide the pedigree of data values. 

The repository should also: be controlled to comply with export control restrictions, include data 
to support technology and system level risk analysis, associate a degree of confidence or 
uncertainty with each piece of data, and be able to export a subset of the data for use in other 
applications. The data migration path begins with ATLAS and the TTB initiated in Excel and 
MySQL. The next step is the production of the SBA information model. The follow-on to these 
activities will be in the fonn of experiments evaluating the capability to transfer data between 
MySQL and the SBA database management system. 



DD&I working group participants plan the future of 
the TTB and other technology’ databases 
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4. RECOMMENDATIONS 


Each working group developed recommendations based on its findings. These recommendations 
are summarized below. The full outbrief from each working group can be found in Appendix B. 

4.1 Technology Data Development and Collection Working Group 

The Technology Data Development and Collection Working Group (TDDC WG) technologists 
recommended a number of changes or enhancements to the TTB. (Figure 4 depicts the current 
layout of the TTB.) 

The group grappled with the problem of defining performance and operations metrics for a 
technology without knowing the specific application for the technology. Some components may 
have the same metrics regardless of the surrounding system, but many subsystem performance 
and operating metrics will depend on the system. To address this issue, the working group 
recommended that the TTB include ground rules and assumptions at the three-digit WBS level. 
This would provide a general description of potential applications to technologist and technology 
selection guidance for the system modelers. The working group also recommended that a process 
be developed for dealing with conflicting data areas, and that these data sets need to be examined 
and then judged to be unique from one another or truly conflicting data. It was recommended 
that for the near term judgment should err on side of allowing each conflicting data set to 
become a unique data record. Once teams have been assigned to a given technology, individual 
data records can be better reconciled to condense redundant data sets. 

The group addressed the fundamental question of how to efficiently use technologists’ time to 
collect data, given the more than one hundred thousand data points the TTB is ultimately slated 
to contain. The consensus was the data collection process should involve a core membership yet 
invite new participants to ensure fresh data. The group discussed how to identify the right 
individuals to participate in the process and identified several interdisciplinary or inter- 
organizational strategies. For example, the group suggested including members crosscutting 
technology development teams within NASA in the data collection process. The group also 
noted that if the core membership of the data collection team included both NASA and industry 
members, data verification and definitional issues would be easier to overcome. Finally, the 
TDDC group recommended that the future process discussions involve both users and developers 
of ATFAS models so that architecture and model perspectives could be added to specific 
decisions regarding data collection techniques. 
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Columns represent 12 timeframes 
starting in 2005 and extending 33 
years in 3 year increments. 
Technologists estimate the 
improvement in performance and 
operations and the funding 
required to mature the technology. 
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Predetermined 
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Predetermined 
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Type - TBD 


Figure 4 An example of a record with data and format descriptions as it would appear in the TTB 


4.2 Architectures and Systems Working Group 

Recommendations from the Architectures and Systems Working Group (A&S WG) for future 
ATLAS and TTB activities addressed model development, documentation, maintenance, 
management, and validation, and workshop processes. The group discussed current models 
within the ATLAS library, which include parametric sizing equations related to propulsion, solar 
power generation, power management and distribution, and in-space energy storage. These 
technologies were used because the system model developers considered these technologies to be 
relevant to sizing the system and data was available in the TTB; in addition, many of these 
models were relatively small development activities. As a result, some models may not include 
important parameters. For example, models in the current version of the ATLAS library do not 
use material properties, which significantly affect system mass. The group recommended that as 
ATLAS development activities increase, models be reviewed and, if needed, revised to ensure 
that they include parameters for the most significant technologies and that new models being 
developed incorporate these parameters. 
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Significant technology areas for future models (and by extension, the TTB) included information 
technology, cross-cutting technologies, and software technology that affects operations. Each is 
discussed below. 

System and infrastructure models should emphasize the impact of infonnation technology (IT). 
For example, technologies associated with the TTB Work Breakdown Structure (WBS) 2.5.1 
“Mobile Surface Systems” and 2.5.2 “Subsurface Access” should include IT. Experts at Ames 
Research Center (ARC), Jet Propulsion Laboratory (JPL) and Johnson Space Center (JSC) could 
provide technology data and suggesting additional metrics. Life cycle and mission phases that 
benefit from IT include development, operations and flight crew time. Costs for IT has become a 
significant percentage of the development phase for aeronautic and space systems. Through 
improved IT, operations costs for future exploration systems could be lower than the Space 
Shuttle and Space Station. A major goal of the exploration initiative is to make the flight crew 
less dependent on the ground. Subsystems that require human attention (such as propulsion, 
guidance navigation and control, life support, extra vehicular activity, and others) take astronaut 
time away from science and exploration; consequently, flight crew time requirements will figure 
prominently in trades concerning onboard systems. 

Cross-cutting technologies are not as well represented in the current database as they could be. 
These include technologies such as integrated system health management and software 
verification and validation, which could decrease failure rates, cost overruns, and maintenance 
costs across many systems. Other cross-cutting technologies include fundamental hardware 
advances, such as novel chip technology, micro electro-mechanical systems, and 
nanotechnology, which could have implication in sensing, mass, and reliability across many 
systems. 

Future system models should include parametric equations for technologies that affect reliability, 
safety, and cost. These include software affecting operations such as: planners and schedulers, 
system health management software, autonomous systems, training and simulation tools, and 
data analysis and management software. Many of these software technologies reduce the size of 
the ground system workforces and increase the time available for a flight crew to conduct 
science and exploration, thereby increasing mission efficiencies. Other safety, reliability, and 
cost recommendations were that transportation system models include abort and retum-to-Earth 
capabilities, and also, that system models include technology options that affect maintainability 
and human factors. Finally, the group noted that defining the cost estimating relationships for 
“ilities” is an interdisciplinary process that must involve the technologists that define the 
performance and operations metrics of specific components or subsystems, the domain “ility” 
expert, the system modeler, and the cost analyst. 

The working group also identified other desirable model features (in addition to the inclusion of 
particular technologies or types of technology. The group recommended a report that 
summarizes user inputs; automated notification of TTB updates; and a flag to indicate a user- 
defined technology. The team recommended that the ATLAS project produce development 
roadmaps that identify model content and number of users, and provide web-based presentations 
to new users; that future TTB developments include fields for data entry dates and sources; and 
that additional TTB fields for assessment of quality or confidence in the data be added. 
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The A&S working group recommended that the documentation developed with the ATLAS 
prototype include: requirements, a detailed design document, a user’s guide, a model developer 
guide, a test plan, and a system model compendium. The group recommended that each system 
model include documentation worksheets such as an introduction, model information, and a 
change log. Future versions of ATLAS system models should be required to include 
documentation of the algorithms, descriptions of technology parameters used by the models, and 
information worksheets to explain case studies and their assumptions about the space exploration 
architecture. 



Figure 5 A notional process for data collection 
and model integration 


The working group identified 
maintenance, configuration 
control, and validation as potential 
issues for ATLAS system models. 
One specific insight was that 
maintaining system models and 
data within the TTB may require 
on-going support contracts that can 
provide training, trouble-shooting, 
update documentation, and 
facilitate the integration process for 
new models and data. The working 
group recommended implementing 
configuration control and 
validation processes. The 
validation process could involve 
modeling a known architecture and 
mission campaign, for example, 
Apollo. Cost analysis could 
compare results from ATLAS with 
historical data to verify the system 
and validate the models and data. 


Comparisons between results from ATLAS and SBA or other analytical tools could support a 
validation process. After passing a verification and validation process, a control board could 
stamp the system model with a logo that indicates approval. Figure 5 depicts a notional process 
for collecting and verifying data in conjunction with developing and integrating models that use 
the data. On the left side of Figure 5, the ATLAS team works with representatives to collect data 
from the technology development community and enter the data into the TTB. In parallel, the 
system modeling community produces Excel workbooks that the ATLAS development team 
integrates into ATLAS. Using data from the TTB, the ATLAS team tests the models and 
presents the results at a TTB TIM. Concurrently, the TTB team conducts a data verification 
process with the technologists and presents the results at the same TTB TIM. As the 
technologists and systems engineers review the results of the ATLAS model testing, they should 
reach a consensus on the validity of the data and models. Recommendations from the TIMs feed 
back into the TTB and improve the data collection and verification processes. 


Finally, the group suggested that briefing the results of previous relevant TIMs at the opening of 
each new TIM, sharing the details of the ATLAS development process and project plan with 
those working on ATLAS related activities, and developing tools to enable on-going 
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communications with the communities of interest in ATLAS and the TTB. Existing studies and 
study efforts that could benefit the development of ATLAS system and operations models. 
Examples given were: a FY 2003 study by the Intelligent Systems program of the impact of 
different classes of infonnation technology on the effectiveness of Mars Rover missions; an in- 
progress ASTP study on autonomy for NASA Crew Exploration Vehicle (CEV) operations is 
producing rough estimates of how stations and shuttle operations costs decompose by major 
system, and rough estimates of the development costs for automation; and, an ongoing activity 
with funded by the Software and Intelligent Systems Modeling program evaluating the impact of 
some classes of integrated systems health management technology on CEV operations. 


The DD&I working group recommended that ASCT establish a verification and validation team 


data requirements in the statements of work of contracts awarded by the ESR&T programs. The 
group suggested that data requirements identify the TTB data format and indicate that the 
termination review process for each contract year will require evidence that the contractor 
provided technology data in that format. The group also urged that formats of the technology 
roadmaps be consistent between the ATLAS TTB and campaign analysis data requirements. 

Figure 6 depicts the working relationships between the Simulation Based Acquisition (SBA) 
development team and the ATLAS development team. During 2005, the ATLAS team will 
continue development of a web-based version of the TTB while the SBA develops the NASA 
Exploration Information Ontology Model (NeXIOM). The SBA team will incorporate the data 


4.3 Database Definitions and Interfaces Working Group 



to define requirements for 
the TTB data collection 
process. As for 
technology database 
development, the group 
recommended that 
ATLAS project continue 
with the development of 
Excel and MySQL 
versions of the TTB, 
while a concurrent 
systems-based acquisition 
effort develops an 
information model that 
incorporates the TTB 
ontology. The working 
group also recommended 
that the ATLAS team 
adopt a risk analysis tool 
instead of developing one 
as a part of ATLAS. 


Figure 6 Relationships between ATLAS and SBA 


The group recommended 
improving technology 
data collection by placing 
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formats from the TTB into NeXIOM. The plan discussed by the group calls for experiments 
conducted in May 2005 that will evaluate the capability to transfer data between the two 
databases. A successful demonstration of NeXIOM will motivate a migration of data from the 
TTB to NeXIOM. Risk analysis constitutes another area where the ATLAS and SBA teams can 
work together. Figure 6 illustrates how the SBA team will develop risk analysis tools and the 
ATLAS team will include the tools in the ATLAS tool suite. In Phase II of ATLAS 
development, the SBA team can incorporate the ATLAS tool suite into the SBA environment. 

4.4 Recommendations from Plenary Session Discussions 

The meeting concluded on day three with a plenary session. Working group leaders briefed each 
group’s accomplishments and recommendations. In addition, an open discussion among the 
meeting attendees identified potential improvements to the data collection and workshop process. 
Finally, Doug Craig closed the meeting with remarks that provided an overview from the 
perspective from NASA Headquarters. 

Several recommendations were made in the general discussion, particularly regarding future 
workshops. As the ATLAS team takes the next steps it was recommended that workshops be 
used as a forum for reporting on progress and addressing crosscutting issues. Specific 
recommendations for future workshops included: dedicating a day for mapping technologies to 
system models; using the first day of the workshop for model developers to present an overview 
of their models and technologists to present an overview of the technologies; and, using the 
second day to bring technologists and model developers together to identify parametric sizing 
equations and data requirements. It was also recommended that pre-workshop data collection 
activities involve web-based application sharing programs where teams could view the ATLAS 
models and contents of the TTB from a live, shared file. This would allow the participants to 
already be familiar with the data and the models that use the data before they arrived at the 
workshop. 
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5. NEXT STEPS 


Near-term actions following the TIM address TTB processes for access and data collection. 

After the final plenary session, Doug Craig met with members of the ATLAS team and the 
ASCT leadership team to discuss the TIM and formulate near-term plans, including identifying 
NASA technologists to participate in the data collection process, defining data requirements for 
contracts, and documenting a procedure for establishing accounts in the web-based TTB. A 
process was laid out and actions assigned. 

Headquarters will identify NASA technologists who will in turn identify other NASA 
technologists within their area of expertise. Technologists working on an ESRT contract will 
receive requirements to provide data for the TTB. The ATLAS development team will establish 
a process for creating accounts on the web-based version of the TTB. After establishing TTB 
accounts for the technologists, the ATLAS team will conduct interviews with each technologist 
and provide documentation on the fields of the TTB database. Through these interviews, the 
ATLAS team will collect the technology data and invite the technologists to review the data in 
the TTB. The data collection and vetting process will incorporate many of the recommendations 
from the TIM reflected in this report. Agendas for future TTB TIMs will facilitate discussions 
between the ATLAS system model developers and the community of technologists so that each 
group will understand the needs of the other. 

In 2005, the ATLAS development team will conduct a data collection and vetting process for the 
TTB, develop system models for the ATLAS library, and produce documentation that explains 
these processes. If authorized to proceed into Phase II in 2006 and beyond, the ATLAS team will 
conduct a number of training workshops to recruit system model developers and technologists to 
populate the TTB. Each year, the ATLAS team will host two TTB TIMs to facilitate 
communication between the modeling and technology communities. Figure 7 depicts the spiral 
development process of ATLAS and the recurring TTB TIMs. 
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Phase II 


FY 2008 


FY 2007 


FY 2006 


Phase I 


FY 2005 


ATLAS Unleashed 

• Improved Models 

• Engineering Process 

• Modeling Community 

Architecture Optimizer 

• Optimizer Tool(s) 

• Engineering Process 

• Lessons Learned 

System Library 

• Training Materials 

• System Models 

• Technology Data 

ATLAS Tool Suite 
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• Database 
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• Data Visualizer 
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System Modelers 
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Figure 7 Spiral development lifecycle for ATLAS 
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APPENDIX B— WORKING GROUP OUT-BRIEFS 




Technology Data Development & Collection 
Working Group 

Tuesday November 9th 

Chairs: Jaime Esper & Monica Doyle 
Rapporteurs: Elaine Gresham and Carie Mullins 

November 8, 9, and 10, 2004 1 


Charge to the Working Groups: 
Technology Data Development & Collection 

* Original Objectives: 

- Review & come to consensus on data from group participants 

• Operating assumptions need to be refined prior to reaching a consensus 

• Developed operating assumptions that will form the basis for this consensus 

- Define growth scenarios for selected data set 

• Also dependent on operating assumptions 

- Identify person/source to complete blank fields 

• Done for many WBS elements, expecting additional inputs 

- Develop completion plan for marginal data sets, with likely assignments 

• To be tasked by respective technology leads 

• Products 

- Completed data sets, with growth figures - TBD 

- Assignment list for follow-up data for workshop participants - see above 

- Sources for continuing data collection 

• Have identified potential sources for future data collection 

• Who will fund this? 

- Working group out brief 


November 8, 9, and 10, 2004 2 
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Participants 




• Participants: 


Jaime Esper (Chair) 

NASA GSFC 

Monica Doyle (Chair) 

SAIC 

Carie Mullins 

The Tauri Group 

Elaine Gresham 

The Tauri Group 

Steven Skonieczny 

ISSI 

Jim Sanders 

ISSI 

Jim Thomas 

ISSI 

Tim Sarver-Verhey 

NASA GRC 

James Crawford 

NASA ARC 

Al Conde 

NASA SSC 

Darby Magruder 

NASA JSC 

Arthur Bradley 

NASA LARC 

Kathy Gavitt 

NGC 

Hobson Lane 

NGC 

Gene Rogers 

Boeing 

Jeff Elbel 

SAIC 

David O’Neil 

QTEC 

David Smitherman 

NASA MSFC 

LaMonte Dent 

UAH 

Don Perkinson 

Sverdrup 


November 8, 9, and 10, 2004 
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Accomplishments 


• Total number of 
technology options where 
data was added 

- Collected 27 data sets 
covering 16 Level 3 WBS 
areas 

• Identified data collection 
working group members 

• Discussed methods for 
improving data collection 
and validation 

• Specific WBS areas 
addressed: 

- 2.2.1 Solar power 
generation 

- 2.6.1 Earth-to-orbit 
transportation 



Received Data Sets 


November 8, 9, and 10, 2004 
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Data Collection Recommendations 


* Conflicting data areas 

- Not all data sets received have been examined in any detail 

- Identified complimentary data sets based on differing operating assumptions 

* Recommendations for future data collection 

- Better define ground rules and assumptions regarding technology needs, 
context for metrics. 

- Vet existing data through independent reviewers 

- Bring “Fresh Blood” to the technology workshops, but maintain core 
participation 

- Bring more users and modelers to review for more integrated discussions. 

* Lack of proprietary data constrains the utility of the database. 

* ITAR restrictions require password protection which limits users 
and impedes international participation 

- this will be an issue if/when the TTB becomes a broader database, i.e. 
beyond the scope of the ATLAS model 


November 8, 9, and 10, 2004 5 



Operating Assumptions 

* Lack of systems engineering requirements definition necessitates 
development of constraints and operating assumptions by 
technologists. 

- Initial inputs resulted in inconsistent data sets which were split into multiple 
records. 

* Define a set of operating assumptions that will allow technologists 
to provide consistent data 

- Operating assumptions are technology-dependent and should be defined for 
each Level 3 WBS (possibly at a lower level). 

- A set of simplifying assumptions (for 1 st cut) include: 

• BOL performance numbers for all technologies 

• Environment (1 AU) 

• Vacuum (GEO environment); Atmospheric (Mars or Earth) 

• What ancillary technologies are included (if any), ex: deployment mechanisms, etc. 

* Some technology records will be defined by technologies + 
configuration. Examples: 

- flexible vs rigid Single Crystal MBG arrays are separate technology options 

- Need to provide precise definitions of the tech options, the assumptions and 
the data collection process prior to asking for data estimates. 


November 8, 9, and 10, 2004 6 
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Data Collection Process 

• For an efficient data collection process, we need to... 

- Define the technology options very specifically 

- Create an ongoing group of technologists that are updated on the TTB 
progress and that will keep the TTB updated with changes in technologies 

- Identify subject matter experts to assign to the TTB (level 3 areas) to 
coordinate assumptions, and data collection 

- Define the assumptions by technology 

- Future participation and contributors 

• Users of ATLAS participate in data collection working group (users and modelers 
and program managers) 

• Work with NASA cross-center working groups (for example low-thrust high ISP) 

• Cross-industry working groups possibly the AIAA technical committees to provide 
contacts 

• Define the scope of work that is needed from contractors and NASA to allow them to 
plan future participation 

* If multiple, disparate data sets are received and cannot be 
reconciled, we should retain all data sets as separate records. 

- This is ok as a first cut since we may not get multiple data sets. 

- As # of data sets increases, we may need to define data ranges. 


November 8, 9, and 10, 2004 7 



Propulsion Data Collection 


• Propulsion Data Collection - specific issues 

- ETO is chemical propulsion only 

- In space propulsion includes a range of technologies (chemical, 
electric, NEP, nuclear thermal, etc.) 

- Difficult/impossible to fill in data for advanced engines when mission 
requirements are not defined. 

• there would be arbitrary-ness involved if technologists are asked to 
provide data for advanced propulsion concepts. 

• may need to be more application dependent 

• need to know vehicle, power source, thrust req, etc. 

- Develop a set of case scenarios to define a number of point designs 
for the TTB 

- TRL 6 needs to be reached 6 years prior to use - assuming we start 
today, the earliest this database can be valid for propulsion is the 
2014 time frame 

- Landing propulsion system driver of in-space propulsion 
development (2.6.3 could add information in about specific landing 
technologies) 


November 8, 9, and 10, 2004 8 
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Goal & Threshold Values 


• How do we define “Goal” and “Threshold” metric values? 

- Option 1: replace these designations with “mean” and “deviation” of 
the probability distribution function 

• as a 1 st cut, we can assume a normal distribution throughout. Note that 
distributions are not yet implemented. 

- Option 2: replace these designations with “Min” and “Max” and use 
these values to bound the data received. 

• Issue: Given multiple data sets for the same technology, storing Min 
value and Max value of each of each metric in the Min and Max columns, 
resp., may result in a data set that’s not physically realizable. 

• For Timeframe 0 (state-of-the-art, FY05), 

- A single value should be recorded (Goal=Threshold) 

- If multiple data sets are received for a single technology, try to 
reconcile any discrepancies. 

- If these cannot be reconciled, select the “best” technology. 


November 8, 9, and 10, 2004 9 



Technology Readiness Level 


• TRLs should not decrease within a record 

- if technology improvement results in a drop in TRL, a new 
technology record should be created. This new technology record 
begins at the point at which the TRL decreased. 

- TRLs can stay the same or increase for a given technology 

• TRL progression path(s) needs to be determined before 
technologists can provide data projections 

- Technology maturation path: Technology performance metrics may 
stop improving while the maturity is advanced to TRL 6. 

- Pure Research Path: Technology performance improves while TRL 
may not increase. Technology may remain at breadboard level. 

- Should we include both and use these to interpolate? 

- Can Spirals be used to assign TRLs of 6 or higher? These may 
determine when some technologies need to be at TRL 6. 


November 8, 9, and 10, 2004 10 
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Architectures and Systems 
Working Group 

Tuesday November 9th 


Chair: Harvey Feingold 
Rapporteurs: Carissa Christensen 


November 8, 9, and 10, 2004 


1 


Charge to the Working Groups: 
Architectures and Systems 

• Objectives 

- Define what is needed for modeling and analysis 

- Discuss existing and planned models, and technology data used 

- Discuss important technical data that should be included in models 

- Discuss architecture analysis for TTB, ATLAS, and SBA 

- Identify strategies for integrated use of TTB 

• Products 

- Working group out brief 

- System to technology mapping 


November 8, 9, and 10, 2004 2 
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Participants 



• Clara Welch 

• Carissa Christensen 

• Bill Sampson 

• David Reeves 

• Daniel Dean 

• Steve Patrick 

• Steve Hoffman 

• James Lewis 

• Mark Gerry 

• Ramona Cummings 

• Craig Allsop 

• Tony Gross 

• Steve Skonieczuys 

• Harvey Feingold 

• Wayne Goode 


November 8, 9, and 10, 2004 
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Accomplishments 


• Findings 

- Context 

- Users 

- Functions 

- Optimization 

• Recommendations 

- Future Model Development Management 

- Documentation 

- Maintenance 

- Management 

- Validation 

- Process 


November 8, 9, and 10, 2004 4 
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Findings 


November 8, 9, and 10, 2004 
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Context 


• ATLAS = ‘quick look’ assessment of what makes sense 

- Compare technologies, identify useful technologies 

- Top-level architectural trade studies 

- “Living forecast” of long-term missions/plans, enabling consideration 
of technology insertion of changing technologies 

• Trades 

- ATLAS (top level) 

- Other models and analyses (detailed) 

• Engineering modeling (TTB should be fed by this level, but 
ATLAS is not appropriate for detailed engineering analysis) 


November 8, 9, and 10, 2004 6 
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Function of ATLAS 


* Determine if there is a long-term technology payoff 

* Not an engineering design tool - but, what is expectation of ATLAS 
as part of SBA 

* Point of model is not to analyze systems or design missions; 
purpose is to compare different technologies on mission strategies 
or campaign strategies 

- Will this statement always be made? Will model continue to be used 
correctly? 

* Who are the users? 

- John Mankins originally was target user; changing (growing) user population 

- Knowledge of user; need for documentation, user friendliness, error checking 
- GIGO could be damaging 

* How will it be used? 

- Modelers 

- Technologists, independent of modeling 


November 8, 9, and 10, 2004 7 




Users 


• Who should ATLAS be designed for? 

- Managers and planners 

• Headquarters program managers 

• Strategic planners 

• Program planners 

- Engineers, technologists, mission designers 

- Students (student version as a heuristic tool?) 

- Input to SBA 

• ATLAS funding for technologies 

• Input into requirements development 

- Pick best technologies 

- Prototyping 

- Deployment 

• TTB may feed whole life cycle of SBA 

• Who else might use ATLAS results? 

- Budget analysts 

- External stakeholders (OMB, Congress) 

- Contractors 

- Academic community - curriculum, grants, studies 

• TTB may be the most useful element of ATLAS for a broad base of 
users 

November 8, 9, and 10, 2004 8 
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Optimization 


• Planned as part of future development 

• Automated, smart optimization 

• Figures of merit 

- Cost 

- Mass 

- Safety, risk 

• Vary 

- Technology 

- Systems 

- Architectures 


November 8, 9, and 10, 2004 9 



SBA Relationship to ATLAS 


• Data dictionary/ontology for technology across databases 

• SBA 

• Engineering analysis environment 

• Support strategy to task activity; traceability from top objectives to specific technology 

• Model different architectures to do sensitivity analyses on different capabilities 

• Enable CE&R contractors to run their models in the environment 

• Contractors respond to RFPs with models in correct format 

• Who is doing what level of analysis? 

- SBA is lower level; ATLAS is higher level strategic analysis 

• SBA will have a ‘point of departure’ model - baseline to judge others against 

• Requirements group will develop 1, ‘we’ will develop another; range of results 

• CER models will in formats that can use ATLAS - esp TTB 

- Plan is to translate SBA models into ATLAS compatible formats 

- Competition sensitive technology information 

- Models into right format 

- SBA has a risk tool - can that be plugged into ATLAS? 

• Data validation scale (estimate vs test results) 

• Validation of ATLAS model through use of SBA to evaluate proposed systems as part 
of acquisitions 

• Users 

- Campaign assessment 

- SBA modelers 


November 8, 9, and 10, 2004 10 
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Recommendations 


November 8, 9, and 10, 2004 


Future Model Development: 
Enhancement of Existing Models 


* Current models already include (though not comprehensively): 

- Propulsion 

- SPG 

- PMAD 

- In-space energy storage 

- Why included? 

• Modelers viewed as relevant 

• Data available in TTB 

- These models have many gaps in technologies that are included 

• Step 1 is to scrub these models to ensure that they have the most significant 
technology drivers in them 

• Revise models 

* Models do not include many other technologies that affect mass, 
such as... 

- Materials properties 


November 8, 9, and 10, 2004 12 
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Future Model Development: 
Include Technologies Affecting Reliability, Safety, Cost 


There are technologies the models do not reflect, even in some cases where there is data in the TTB 

- Affect reliability, safety 

- These don’t significantly affect mass but do affect cost 

Software examples 

- Planners and schedulers 

- System health management 

- Human assist technologies to aid astronauts, ground operators (advisory systems, long-term monitoring, voice interface) 

- Intelligent support for automation and rover systems (can supply a high-level goal and system will decide how to achieve 
goal) 

- acttv'tfes l^ccorcfing'fyfnagement tools (will affect mission ops by partially analyzing data from sensors and changing mission 

Autonomous flight management and operations 

- Software and hardware 

Processor power 

- Cost driver (rad hardening) 

Simulation and training of humans (esp for EVA) 

- T echnologies for simulation systems 

- Technologies for training (faster, better) 

Crew safety 

- Abort 

- Return to Earth 

- Human rating-related technologies, testing, etc 

Other important information/capabilities for models 

- Maintainability (esp self-maintainability) 

• Software maintainability 

• Hardware maintainability 

- Human factors (e.g. fatigue on long-term missions) 
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Future Model Development: 
Functionality 

• Printout of inputs, outputs, and appropriate flags 

- Summary of choices 

- Flag of what has been changed from one case to another 

- Flag that results include ‘user defined’ technology 

- Results always linked to key inputs 

• Reference model 

• Changes from reference 

- FLAG ANYTHING THAT IS DIFFERENT FROM APPROVED MODEL 

- Configuration control and security - approved version 

• Pilot users? 

- Web-based meetings 

- Different points of view 

• Roadmaps? 

- Number of models, status, content 

- Users 

• Need one graphic of ATLAS (in, out) (CEO chart) 

• User defined results 

- Charts and graphs 

- Sensitivity analyses Notification of TTB update to users with active models 


November 8, 9, and 10, 2004 14 



31 


Future Model Development: 
New Model Topics 

• Operations models considering impacts of e.g. automation 

• Surface operations 

• Mars missions (artificial gravity, shielding) 

• Lunar, Mars isru 
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Future TTB Development 


• TTB TBD - more robust default for missing information 

• Column in TTB for date, source 

• TTB data 

- TTB should show currency of data 

- Control of data 

- WBS easy to understand 

- SBA will require long-term data to support integrated analysis of 
systems and systems-of-systems; ATLAS will need to have a flexible 
architecture to bring in real-world elements 

- Reliability data? 

• Reliability trades against mass, eg; different reliabilities = different 
technologoies? 

• Gross assessment of reliability? (h,m,l)? 

• Note: reliability is associated with a mature technology 
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Documentation 



• Existing 

- Model introduction page - model developer, dates, etc 

- Initial versions of guide and plan documents 

• User’s guide 

• Model developer’s guide 

• Test plan (ATLAS overall) 

• System model compendium (partial; system definition, description of model, constraints and 
applicability of model) 

• Planned 

- Completion of guide and plan documents 

• Needed 

- Documentation of algorithms 

• Documentation of algorithms and logic will be necessary for validation process 

- Separate documentation as well as embedded; provide these instructions in developer’s 
guide; standardize documentation approach 

- Model functions 

• Print out technologies used (tech table) 

• Label different cases being compared 

- Case study introduction page in case study electronic file 

- Documentation - define common interfaces for inputs and outputs to enable 
compatibility with other models 
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Maintenance 


• Data 

• Software 

• Documentation maintenance 

• Training/support 

• Long-term support 

- Implications of BAAs, new approach of Code T, connection to 
Simulation-Based Acquisition (SBA) 
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Configuration Control 


• Configuration control board (CCB) 

- Models 

- Technology inputs 

- Process 

- Documentation 
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Validation: 

How should we determine if models are acceptable? 


Trusted models 

- Experience 

- Reference to real-life situations 

• Apollo as case study for ATLAS 

• Company (Q-Tech) is retrieving data on Apollo, Shuttle, Skylab 

- Takes a long time to build confidence in complex models 

- Planned and frequent model check procedures 

• Core model 

• System model 

Many of ATLAS models are based on concepts that do not exist; can’t test based on 
known values 

- Experience with models, modelers, modeling approach 

- Sanity check/peer review 

- Existing model development community is fairly closed 
Validate ATLAS with detailed design studies? 

- ATLAS results for a mission compared to a detailed design study 

- What is a good design study? 

- Validation vs correlation 
What is “good enough”? 

- Not a detailed design tool; shouldn’t be expected to come up with the same answers as a detailed 
design tool 

- Needs to produce RELATIVE answers among a few choices; does not need to produce ABSOLUTE 
answers 
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Validation: 
Trusted Model Seal of Approval 


• Who determines? 

- Review board 

- Independent 

- Ad hoc for each model 

- Informal in that model could be sent to individuals; who would answer specific set of questions; quick top-level look 

- Criticisms/response 

- Final approval or disapproval by ? 

• What is process? 

- Algorithms 

- Input, output 

- Compliance with ATLAS requirements 

- NASA Independent Program Assessment Organization 

• Run from Langley 

• Experts from across field centers 

• Designed to be a quick investigation 

• Model by model? Overall program commitment? 

• Critical issue: how define process 

- Software validation and verification process within NASA? Is this a fit for ATLAS models? 

- Code T process owner 

- Documentation of algorithms and logic will be necessary for validation process 

• How indicated? “Seal of approval?” Excel add-in? logo provided? 

• What criteria? 

- Technology data current within defined period; technology data from TTB (which has some process for accepting data) 

• Specifies purpose of model 
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Caution 

Documentation, Validation, Management 


• But... don’t want to lose flexibility, speed 

- How? HARD BOUNDARIES regarding how it is used 

- Keep models simple and quick 

- Versatility - easy to use, accessible 
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Process 



• Results of previous workshops should be briefed at each 
workshop 

• Share details of program plan 

• On-going communication with community of interest 
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Database Definitions and Interfaces 
Working Group 

Tuesday November 9th 


Chair: Doug Craig 
Rapporteurs: Dan O’Neil 
Emily Horton 
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Charge to the Working Groups: 
Definitions and Interfaces 


• Objectives 

- Develop analysis plan for ESR&T 

- Asses possibility for use of TTB by other tools 

- Asses possibility for use of ATLAS by other tools 

- Tools workshop pre-planning 

• Products 

- Working group out brief 

- Draft analysis plan for ESR&T 
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Participants 



• Participants 

- Doug Craig 

- Steve Cavanaugh 

- Jay Falker 

- Pat Troutman 

- Mark Shirley 

- Jeremy Vander Kam 

- Ravi Deo 

- Emily Horton 
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Definitions 


• Architecture - A collection of systems that provide the capability to 
perform one or more types of missions 

• Campaign Analysis - Evaluates the benefits across a sequence of 
missions and systems. Investigates the sensitivities of campaigns to 
technology portfolios. 

- Supports the ASCT Advanced Study teams 

- Answers action items received by the ESR&T programs 

- Annual technology assessment and investment process 

- Verifies and validates technology data. 

• Point-of-Departure (PoD) Architectures - Produced by NASA and 
the Concept Exploration and Refinements (CE&R) Teams as a 
reference for benefit assessments. 

• Ontology - An organizational structure for data including the 
technology data. 

• Threshold value - The minimum achievable value within a range of a 
technology parameter. 
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Tools workshop pre-planning 


• Advanced Concepts 

- Pat Troutman document process and products. 

• Technology Database(s) 

- Jeremy Vander Kam and Dan O’Neil document process 

• Campaign Analysis 

- Dan O’Neil document ATLAS output charts and provide to Steve 
Cavanaugh 

• Risk Analysis 

- Jay Falker document the risk analysis activities and tool selection 
process. 

• Data Visualization 

- Doug Craig document chart requirements in concert with SBA. 


November 8, 9, and 10, 2004 5 




39 


Analysis Plan 



• Data Requirements - A verification & validation team within ASCT will define the 
requirements for the TTB data collection. These requirements will define the data 
pedigree. 


• Develop a centralized repository of technology data. 

- Controlled access to the technology records 

- Record owners will provide the pedigree for record data values. 

- While data within records may not be subject to EAR/ITAR export control, the database as a 
whole must be controlled. 


- The database shall include data to support technology and system level risk analysis 

- A field in the database shall express the confidence in the data. Sometimes, degree of 
uncertainty becomes more important than the data. 

- An export function shall provide the capability to extract a subset of the database. 

• Data Migration Path 

- In the near term, Excel and MySQL will instantiate the ATLAS TTB. 

- The SBA team will produce a Windchill technology database. 

- Experiments conducted in May 2005 will evaluate the capability to transfer data between 
MySQL and Windchill. 

• Risk Tool - Evaluate the Freddie Douglas Safety-of- Flight risk analysis tool. By 
February, ESR&T will decide on the risk analysis tool. 
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Action items 



• Jimi Crawford manages a project to obtain operations data from 
United Space Alliance (USA). Establish a relationship between 
the ATLAS activity and this project 

• Produce a list of technologists who will populate the Technology 
Tool Box (TTB) and own tne records. 

• Define working relationships between the ASCT technology 
database development team and technology working groups 
across the agency. 

• The Simulation Based Acquisition (SBA) team and the ASCT 
campaign and technology analysis groups have overlapping 
charters. 

- Define the Level 0 and Level 1 analysis and associated tools. 

- Clearly define the roles and responsibilities of the two teams. 

• The ASCT will be responsible for populating the technology 
database. 
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Recommendations 


• Operations Data Recommendation: To get technology data 
from the contractors, the data requirements specified on the 
contractors must include data formats for the ESMD technology 
database. 

• Campaign Assessment Team - A team will use ATLAS and 
other tools to analyze the architectures and associated missions. 
They provide comments to the Advanced Studies Teams and 
investment recommendations to the ESR&T management. 

• Technology assessments - The technologists determine the 
state-of-the-art and forecasted metrics and enter the data into the 
technology database. 

- A team will perform sensitivity analysis at the system and technology 
level 

- Sensitivity analysis must be performed across many architectures 
and missions. 

• Format of technology roadmaps should be consistent between 
ATLAS TTB and the Campaign Analysis data requirements. 
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